Method for aligning demonstrated user actions with existing documentation

ABSTRACT

There is provided an apparatus for aligning document actions specified in a documented procedure with demonstrated user actions. The apparatus includes an alignment device for computing an alignment between the document actions and the demonstrated user actions based on at least one similarity metric.

FIELD OF THE INVENTION

The present invention generally relates to documented procedures and,more particularly, to a method and apparatus for aligning steps in adocumented procedure with demonstrated user actions.

BACKGROUND OF THE INVENTION

Technical support procedures are typically very complex. Users oftenhave trouble following printed instruction describing how to performthese procedures.

In a study to investigate how end users perform technical supportprocedures while following printed directions, the mouse and keyboardactions of eleven subjects were captured as they followed five pages ofwritten instructions (including screenshots). These instructions werecopied from IBM's internal technical support website. The instructionsdescribed a procedure to modify and verify the Domain Name Server (DNS)configuration of a laptop computer. Each subject was presented with adifferent initial configuration, with the eleven configurations chosento be distinct and to span the space of possible procedure pathways. Thesubjects were instructed to follow the directions to restore the systemto the correct configuration.

Nearly all participants, regardless of MICROSOFT WINDOWS knowledge, haddifficulties following printed instructions. In particular, it was notedthat people often had difficulty translating from text on the page towidgets on the screen. Moreover, it was noted that people tended to missportions of the instructions, particularly separate portions of textthat described branches of a conditional instruction. Also, it was notedthat people had difficulty following out-of-order instructions andscreenshots, especially those that were not identical to what they sawon the screen.

Accordingly, it would be desirable and highly advantageous to have amethod and apparatus for aligning document actions specified in adocumented procedure with corresponding demonstrated user actions, so asprovide a guided approach to technical support procedures.

SUMMARY OF THE INVENTION

These and other drawbacks and disadvantages of the prior art areaddressed by the present invention, which is directed to a method andapparatus for aligning document actions specified in a documentedprocedure with corresponding demonstrated user actions.

According to an aspect of the present invention, there is provided anapparatus for aligning document actions specified in a documentedprocedure with demonstrated user actions. The apparatus includes analignment device for computing an alignment between the document actionsand the demonstrated user actions based on at least one similaritymetric.

According to another aspect of the present invention, there is providedan apparatus for guiding user actions with respect to a documentedprocedure displayed on a user interface. The documented procedure has aplurality of steps specified therein. The apparatus includes analignment device, a highlighting device, and a scrolling device. Thealignment device is for computing an alignment between the documentactions and the demonstrated user actions based on at least onesimilarity metric. The highlighting device, connected in signalcommunication with the alignment device, is for visually highlighting onthe user interface at least a portion of a step in the documentedprocedure based on the alignment computed by the alignment device. Thescrolling device, connected in signal communication with the alignmentdevice, is for automatically scrolling through each of the plurality ofsteps of the documented procedure on the user interface based on thealignment computed by the alignment device.

According to yet another aspect of the present invention, there isprovided a method for aligning document actions specified in adocumented procedure with demonstrated user actions. The method includesthe step of computing an alignment between the document actions and thedemonstrated user actions based on at least one similarity metric.

According to a further aspect of the present invention, there isprovided a method for guiding user actions with respect to a documentedprocedure displayed on a user interface. The documented procedure has aplurality of steps specified therein. The method includes the steps ofcomputing an alignment between the document actions and the demonstrateduser actions based on at least one similarity metric, visuallyhighlighting on the user interface at least a portion of a step in thedocumented procedure based on the computed alignment, and automaticallyscrolling through each of the plurality of steps of the documentedprocedure on the user interface based on the computed alignment.

These and other aspects, features and advantages of the presentinvention will become apparent from the following detailed descriptionof exemplary embodiments, which is to be read in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood in accordance with thefollowing exemplary figures, in which:

FIG. 1 is a block diagram illustrating a system 100 for aligningdocument actions specified in a documented procedure with correspondingdemonstrated user actions, according to an illustrative embodiment ofthe present invention;

FIG. 2 is a flow diagram illustrating a method for aligning documentactions specified in a documented procedure with correspondingdemonstrated user actions, according to an illustrative embodiment ofthe present invention; and

FIG. 3 is a flow diagram further illustrating step 230 of FIG. 2,according to an illustrative embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention is directed to a method and apparatus for aligningdocument actions specified in a documented procedure with correspondingdemonstrated user actions.

A brief description will now be given of an implementation of thepresent invention to illustrate some of the many attendant advantagesand features of the present invention. According to one illustrativeembodiment of the present invention, a user opens an onlinedocumentation file, such as an operating manual or an IBM Redbook, turnsto a particular documented procedure and, as the user performs theprocedure, the corresponding text and/or graphics are highlighted andadvance (e.g., scrolling to the next step, next page, etc.) along withuser. Optionally, the next step to be taken in the documentation may behighlighted. In the likely event the user is performing the procedureusing a user interface, the highlighting and advancing may be performeddirectly in the user interface.

Thus, in accordance with the present invention, novice users couldbenefit from having annotations explaining the key concepts in theprocedure, and experts could benefit from partial automation that helpsthem avoid mistakes. Several ways have been identified in which aninteractive help system could be an improvement over printedinstructions. For example, in accordance with the principles of thepresent invention, the present invention may advantageously employ thefollowing features to help guide an individual to perform steps in adocumented procedure: automatically highlighting the target widget onthe screen; automatically detecting and branching on conditionals; andproviding a visual indicator of which step the user is currently “up to”in the procedure. It is to be appreciated that the present invention isnot limited to the preceding features to aid a user in performing adocumented procedure and, thus, given the teachings of the presentinvention provided herein, other features may also be employed whilemaintaining the spirit of the present invention.

It is to be appreciated that as used herein the words “highlighting” and“highlight” refer to any visual, audible, and tactile indication or cuethat may be used to identify (or make stand out) an item of interest ina documented procedure. Such items may include, but are not limited to,a step or portion thereof, a word, a phrase, a symbol, a hyperlink, agraphic, and any other item that may be found in a documented procedure.It is to be further appreciated that the present invention is notlimited to the preceding ways in which to identify an item, or to theitems specified and, thus, other ways and other items may be used toprovide “highlighting” as used herein, while maintaining the spirit andscope of the present invention.

The present description illustrates the principles of the presentinvention. It will thus be appreciated that those skilled in the artwill be able to devise various arrangements that, although notexplicitly described or shown herein, embody the principles of theinvention and are included within its spirit and scope.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the principlesof the invention and the concepts contributed by the inventor tofurthering the art, and are to be construed as being without limitationto such specifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, andembodiments of the invention, as well as specific examples thereof, areintended to encompass both structural and functional equivalentsthereof. Additionally, it is intended that such equivalents include bothcurrently known equivalents as well as equivalents developed in thefuture, i.e., any elements developed that perform the same function,regardless of structure.

Thus, for example, it will be appreciated by those skilled in the artthat the block diagrams presented herein represent conceptual views ofillustrative circuitry embodying the principles of the invention.Similarly, it will be appreciated that any flow charts, flow diagrams,state transition diagrams, pseudocode, and the like represent variousprocesses which may be substantially represented in computer readablemedia and so executed by a computer or processor, whether or not suchcomputer or processor is explicitly shown.

The functions of the various elements shown in the figures may beprovided through the use of dedicated hardware as well as hardwarecapable of executing software in association with appropriate software.When provided by a processor, the functions may be provided by a singlededicated processor, by a single shared processor, or by a plurality ofindividual processors, some of which may be shared. Moreover, explicituse of the term “processor” or “controller” should not be construed torefer exclusively to hardware capable of executing software, and mayimplicitly include, without limitation, digital signal processor (“DSP”)hardware, read-only memory (“ROM”) for storing software, random accessmemory (“RAM”), and non-volatile storage.

Other hardware, conventional and/or custom, may also be included.Similarly, any switches shown in the figures are conceptual only. Theirfunction may be carried out through the operation of program logic,through dedicated logic, through the interaction of program control anddedicated logic, or even manually, the particular technique beingselectable by the implementer as more specifically understood from thecontext.

In the claims hereof, any element expressed as a means for performing aspecified function is intended to encompass any way of performing thatfunction including, for example, a) a combination of circuit elementsthat performs that function or b) software in any form, including,therefore, firmware, microcode or the like, combined with appropriatecircuitry for executing that software to perform the function. Theinvention as defined by such claims resides in the fact that thefunctionalities provided by the various recited means are combined andbrought together in the manner which the claims call for. Applicant thusregards any means that can provide those functionalities as equivalentto those shown herein.

FIG. 1 is a block diagram illustrating a system 100 for aligningdocument actions specified in a documented procedure with correspondingdemonstrated user actions, according to an illustrative embodiment ofthe present invention.

The system 100 is intended to cooperate with (or be included in) acomputer 199 that is to be used to perform the documented procedure. Forexample, the computer 199 may simply be a desktop computer, a laptopcomputer, a personal digital assistant, an electronic tablet, and soforth that is capable of displaying a documented procedure for a user toexecute. The computer 199 may optionally have the capability to access anetwork (e.g., the Internet, a company intranet, etc.) to download adocumented procedure. Thus, it is to be appreciated that the documentedprocedure may be located locally or remotely with respect to thecomputer 199. The computer 199 includes one or more processors(hereinafter “processor”) 188, one or more memories (hereinafter“memory”) 187, a network interface 186, and one or more buses(hereinafter “bus”) 185. The computer 199 further includes a userinterface 198 for allowing a user to interact with the documentedprocedure. The user interface 198 may include a keyboard or keypad(hereinafter “keyboard”) 197 and a display 196. The display 196 may be atouch-screen display capable of receiving user inputs via the display.The present invention is not limited to interacting with only computershaving the configuration as shown in FIG. 1 and, thus, other computers,computing devices and corresponding configurations may also be employedin accordance with the present invention while maintaining the spirit ofthe present invention.

The system 100 includes an alignment device 110. The alignment device110 may receive inputs from the user interface 198 or may include acamera (also interchangeably referred to herein as “imaging device”) 111or other device for use in determining demonstrated user actions. Thealignment device 110 preferably further includes one or more processors(hereinafter “processor”) 112, one or more memories (hereinafter“memory”) 113, and one or more buses (hereinafter “bus”) 114 forinterconnecting the elements of the alignment device 110. Additionally,the alignment device 110 may include a step extractor 119 for extractingsteps from a documented procedure. The step extractor 119 may include ascanner and/or other device capable of optical character recognition, aparser, a memory for storing words/phrases/etc. that may be comparedagainst parsed words/phrases/etc., and/or so forth.

The alignment device 110 computes an alignment between the documentactions specified in a documented procedure and correspondingdemonstrated user actions. Based on the computed alignment, a step orportion of a step (e.g., word(s), graphic(s), and so forth) in thedocumented procedure is highlighted so that the user can identify wherehe or she is (e.g., which step is the current step for which a useraction needs to be taken) within the documented procedure. Optionally, anext step may also be highlighted, preferably in a different manner thanthe current step. The alignment may be used to automatically scrollthrough the procedure as the user progresses there through.

It is to be appreciated that the connection configuration depicted inFIG. 1 is merely illustrative and, thus, given the teachings of thepresent invention provided herein, one of ordinary skill in the relatedart will contemplate these and other connection configurations and otherelements that may be employed with respect to a system for aligningdocument actions specified in a documented procedure with correspondingdemonstrated user actions while maintaining the spirit of the presentinvention.

FIG. 2 is a flow diagram illustrating a method for aligning documentactions specified in a documented procedure with correspondingdemonstrated user actions, according to an illustrative embodiment ofthe present invention.

The document is marked up to identify a plurality of document actions(step 205). A document action is an action expected or mandated to betaken with respect to a given document. A list that includes theplurality of document actions is generated (step 210). Steps 205 and 210are preferably performed prior to a user attempting to perform the stepsspecified in the documented procedure.

The next step in the procedure is highlighted (step 215). The systemstate and demonstrated user action are recorded (step 220). An alignmentis computed between the demonstrated user action and the documentactions in the list based on at least one or more similarity metrics(step 230). Based on the alignment, the procedure may be advanced, ifnecessary, and the most likely document action in the procedure (i.e.,the current document action to be taken by the user) is highlighted(step 240).

It is then determined whether or not the user has completed theprocedure (step 250). If the user has not completed the procedure, thenthe method returns to step 215 to highlight the next step in theprocedure. Preferably, the next step is highlighted in a mannerdifferent from that employed with respect to the current step, so as toavoid user confusion. Otherwise, if the user has completed theprocedure, then the method is terminated. It is to be appreciated thathighlighting with respect to the current step and/or the next step mayinvolve highlighting a whole textual and/or graphical description of astep, a portion thereof, an element of the user interface (including,but not limited to, action buttons (e.g., cut, paste, save, etc.), andso forth.

FIG. 3 is a flow diagram further illustrating step 230 of FIG. 2,according to an illustrative embodiment of the present invention. Step230 can also be considered as computing a similarity between auser-demonstrated state/action and steps in a documented procedure(document actions, as specified in the list). According to anillustrative embodiment of the present invention, similarity may bebased on a probabilistic combination of similarity parameters.

The number of words in common between a textual description of ademonstrated user action and document action is determined (step 310).

The similarity of the action types involved (e.g., clicking button andclicking checkbox are more similar than typing) with respect to thedemonstrated user action and the document actions is determined (step320).

A probability distribution (e.g., including, but not limited to, aGaussian distribution) over two or more steps is determined (step 330).In such an approach, the next step in temporal order is usually the mostlikely.

Any common strings in the user interface (e.g., button labels) and thedocumented procedure are determined (step 340).

An alignment based on one or more of steps 310 through 340 is thencalculated (step 350). It is to be appreciated that the alignment mayalso be based on experiences of previous users, using any of the resultsof steps 310 through even 350 for the previous users.

It is to be appreciated that the present invention is not limited to thepreceding alignment or similarity parameters and, thus, given theteachings of the present invention provided herein, other parameters mayalso be employed to determine alignment or similarity while maintainingthe spirit of the present invention.

These and other features and advantages of the present invention may bereadily ascertained by one of ordinary skill in the pertinent art basedon the teachings herein. It is to be understood that the teachings ofthe present invention may be implemented in various forms of hardware,software, firmware, special purpose processors, or combinations thereof.

Most preferably, the teachings of the present invention are implementedas a combination of hardware and software. Moreover, the software ispreferably implemented as an application program tangibly embodied on aprogram storage unit. The application program may be uploaded to, andexecuted by, a machine comprising any suitable architecture. Preferably,the machine is implemented on a computer platform having hardware suchas one or more central processing units (“CPU”), a random access memory(“RAM”), and input/output (“I/O”) interfaces. The computer platform mayalso include an operating system and microinstruction code. The variousprocesses and functions described herein may be either part of themicroinstruction code or part of the application program, or anycombination thereof, which may be executed by a CPU. In addition,various other peripheral units may be connected to the computer platformsuch as an additional data storage unit and a printing unit.

It is to be further understood that, because some of the constituentsystem components and methods depicted in the accompanying drawings arepreferably implemented in software, the actual connections between thesystem components or the process function blocks may differ dependingupon the manner in which the present invention is programmed. Given theteachings herein, one of ordinary skill in the pertinent art will beable to contemplate these and similar implementations or configurationsof the present invention.

Although the illustrative embodiments have been described herein withreference to the accompanying drawings, it is to be understood that thepresent invention is not limited to those precise embodiments, and thatvarious changes and modifications may be effected therein by one ofordinary skill in the pertinent art without departing from the scope orspirit of the present invention. All such changes and modifications areintended to be included within the scope of the present invention as setforth in the appended claims.

1. (canceled)
 2. The apparatus of claim 10, wherein said alignmentdevice identifies a current step in the documented procedure based onthe alignment.
 3. The apparatus of claim 2, wherein the documentedprocedure is displayed on a user interface, and said alignment deviceprovides an identification of the current step to the user interface foruse in highlighting the current step on the user interface.
 4. Theapparatus of claim 2, further comprising a user interface, in signalcommunication with said alignment device, for displaying the documentedprocedure, and wherein said alignment device outputs an identificationof the current step to the user interface, and said user interfacehighlights the current step based on the identification.
 5. Theapparatus of claim 10, wherein said alignment device identifies a nextstep in the documented procedure based on the alignment.
 6. Theapparatus of claim 5, wherein the documented procedure is displayed on auser interface, and said alignment device provides an identification ofthe next step to the user interface for use in highlighting the nextstep on the user interface.
 7. The apparatus of claim 5, furthercomprising a user interface, in signal communication with said alignmentdevice, for displaying the documented procedure, and wherein saidalignment device outputs an identification of the next step to the userinterface, and said user interface highlights the next step based on theidentification.
 8. The apparatus of claim 10, wherein the at least onesimilarity metric is based on at least one of a number of words incommon between a textual description of a demonstrated user action and adocument action, a similarity of an action type involved in both ademonstrated user action and a document action, a probabilitydistribution over at least two steps of the documented procedure.
 9. Theapparatus of claim 10 wherein the demonstrated user actions areperformed with respect to a user interface, and the at least onesimilarity metric is based on common strings in the user interface andthe documented procedure.
 10. An apparatus for aligning document actionsspecified in a documented procedure with demonstrated user actions,comprising: an alignment device for computing an alignment between thedocument actions and the demonstrated user actions based on at least onesimilarity metric; and a camera, connected in signal communication withsaid alignment device, for capturing images of at least some of thedemonstrated user actions, and wherein said alignment device uses atleast some of the captured images to calculate the alignment.
 11. Theapparatus of claim 10 wherein said alignment device computes thealignment further based on previous user experiences with respect toperforming the documented procedure.
 12. The apparatus of claim 10wherein the documented procedure has a plurality of steps specifiedtherein, and the apparatus further comprises a document action extractorfor extracting at least some of the plurality of steps.
 13. (canceled)14. The apparatus of claim 15 wherein the at least one similarity metricis based on at least one of a number of words in common between atextual description of a demonstrated user action and a document action,a similarity of an action type involved in both a demonstrated useraction and a document action, a probability distribution over at leasttwo steps of the documented procedure, and common strings in the userinterface and the documented procedure.
 15. An apparatus for guidinguser actions with respect to a documented procedure displayed on a userinterface, the documented procedure having a plurality of stepsspecified therein, the apparatus comprising: an alignment device forcomputing an alignment between the document actions and the demonstrateduser actions based on at least one similarity metric; a highlightingdevice, connected in signal communication with said alignment device,for visually highlighting on the user interface at least a portion of astep in the documented procedure based on the alignment computed by saidalignment device; a scrolling device, connected in signal communicationwith said alignment device, for automatically scrolling through each ofthe plurality of steps of the documented procedure on the user interfacebased on the alignment computed by said alignment device; and a camera,connected in signal communication with said alignment device, forcapturing images of at least some of the demonstrated user actions, andwherein said alignment device uses at least some of the captured imagesto calculate the alignment.
 16. The apparatus of claim 15 wherein saidalignment device computes the alignment further based on previous userexperiences with respect to performing the documented procedure. 17-32.(canceled)